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Image processing unit and method of associating stored images with current images 



The invention relates to an image processing unit which comprises an input for 
the signal of a current image of a body volume, the body volume being subject to a motion 
comprising several phases of motion^ an input for a signal which represents the phase of 
motion of tiie current image, and a memory in which previous images of the body volimie are 
5 stored together with the associated phases of motion. The invention also relates to a method 
of associating a current image of a body volume, being subject to a motion comprising 
different phases of motion, with a previous image of the body volume. 

The imaging of body volumes is practiced notably in the field of medical 
diagnostics and therapy, that is, in the context of X-ray fluoroscopy. Therefore, the X-ray 

10 projection of a biological body volume will be considered hereinafter by way of example, be 
it that the present invention is by no means restricted thereto and can be used in all fields of 
application with similar secondary conditions. 

A special medical application is formed by the X-ray fluoroscopic observation 
of the propagation of a catheter in the vascular system of a patient The tip of the catheter 

IS should then be advanced as accurately as possible into a region of interest to be treated or 

examined, for example, a stenosis, or a guide wire should be positioned behind said region of 
interest in such a manner that the tip of the catheter is correctly positioned. In this respect it is 
known to display angiographic images of the body volume on a second monitor adjacent the 
current image of the body volume. The angiographic images reproduce the vessels in a 

20 highlighted fashion and can be acquired, for example, by utilizing a contrast agent The still 
angiographic image supports the orientation for the attendant physician as a 'Vascular map" 
or *Yoad map" of the vascidar system. However, the known methods are not suitable to 
achieve a desirable exact association of corresponding locations in the angiogr^hic image 
and the current image, that is, with an accuracy in the miUimeter or sub-millimeter range, 

25 because the body volume of a patient being observed usually is subject to a more or less 
strong (natural) motion which is caused notably by heartbeat and respiration. 

In this respect it is known from DE 199 46 948 Al to acquire in advance 
three-dimensional images of the body organ in different phases of its motion in order to 
determine the position of an instrument, for example, a catheter, relative to a moving body 
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organ, for example, the heart, the phase of motion being determined and described by way of 
an electrocardiogram (ECG) which is recorded in parallel. During the current medical 
intervention the position of the catheter is then measured together with the associated phase 
of the electrocardiogram and the catheter is reproduced in that three-dimensional image 
5 which is associated with the same phase of the electrocardiogram of the body organ. 

However, this method necessitates on the one hand three-dimensional images of the body 
volume and on the other hand does not involve the continuous fluoroscopic monitoring of the 
medical intervention. 

Considering the foregoing it is an object of the present invention to provide a 
10 method and a device which enable stored images of a body volume to be associated with a 
current image with a high precision which is preferably in the millimeter or sub-millimeter 
range. 

This object is achieved by means of an image processing unit which is 
characterized as disclosed in claim 1 as well as by means of a method which is characterized 
IS as disclosed in claim 9. Advantageous embodiments are disclosed in the dependent claims. 

The image processing unit in accordance with the invention comprises: 

- an input for the signal of a "current image" of a body volume, which body volume is 
subject to a inotion involving difif^ent phases of motion. The image may have been 
formed in particular by means of an X-ray apparatus with an X-ray source and an X- 

20 ray detector. However, the image may also have been acquired by means of other 

methods, such as magnetic resonance, ultrasound, scintigrs^hy or the like. The term 
"current image" represents the normal situation in which this image is acquired and 
transferred on-line, that is in real time, be it that applications with "ofiT-line" 
processing should not be excluded. 

25 - An input for a signal which represents the phase of motion of the body volume during 

said current image. 

- A memory in which "previous images" of the body volume are stored together with 
the respective associated phases of motion. The previous images may have been 
acquired by means of the same imaging device which also generates the current 

30 image of the body volume. However, it is also possible for the previous images to 

originate from other sources. In the context of a medical intervention of the kind set 
forth, the previous images may notably be angiographic images in which the vascular 
system in the relevant body volume is represented. 



wo 2004/034329 PCT/IB2003/004339 

3 

The image processing unit is arranged to associate with the current image that 
one of the previous images whose phase of motion is closest to the phase of motion of the 
current image. 

The image processing unit in accordance with the invention enables very 
5 accurate association of a previously stored image of a body volume with a currmt image of 
the body volume, because the phases of motion of the associated images are as close as 
possible to one another, thus minimizing- the motion-induced deviations between the inures 
of the body volume. 

In conformity with a further embodim^t of the image processing unit it is 
10 arranged to determine the distance betwe^i the phases of motion of the current image and the 
associated previous image and to reproduce it for a user. Additionally, or alternatively, the 
image processing unit may also be arranged to determine the time elapsed since the last 
update of the association between current images and a previous image and to reproduce it 
for a user. The reproduction of the distance between the phases of motion and/or the time 
15 elapsed since the last association provides the user with information as to how up to date or 
exact the existing association still is. Therefore, on the basis of this representation the user 
can determine to what extent local correspondences of the images can still be trusted. 

In the context of a special embodiment of the image processing unit, the body 
volume is a biological body volume such as, for example, the heart of a patient, and the 
20 motion of this body volume is caused by the heartbeat and/or the respiration. Heartbeat and 
respiration constitute two important causes of (natural) motions which have an effect 
throughout the body. The special image processing unit is characterized in that the phases of 
motion are described by an electrocardiogram (BCG) and/or the signal of a respiration sensor. 
In conformity with a preferred embodiment of the invention, the image 
25 processing unit is arranged to carry out the following steps of the method: 

calculation of a similarity measure between the current image and a predetermined 
representative image, 
- calculation of the corresponding similarity measures between the representative 
image and the previous images or a sub-quantity of the previoiis images, 
30 - selection of a sub-quantity which contains exactly those previous images whose 
similarity measure relative to the representative image is in a predetermined range 
around the similarity measure of the current image relative to the representative 
image. 
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According to this ^proach, therefore, all previous images are determined 
which are "just as similar or dissimilar" relative to the representative image as the current 
image, that is, all previous images which have the same similarity gap relative to the 
representative image as the current image. Therefore, the sub-quantity which will notably 
5 include also the images which are very similar to the current image. The method has the 

advantage that, aft^ a representative image has been defined, all similarity measures between 
this representative image and the previous images have to be calculated only once after which 
they can be stored. The similarity measure has to be calculated anew again only for the 
ciirrent image. The sub-quantity, therefore, can be defined with little effort during operation. 

10 For the above embodiment of the invention it is necessary to perform a 

suitable definition of the similarity of images or the similarity measure. A preferred similarity 
measure consists in the calculation of the histogram energy of the difference image between 
the two images. In this case a small value of the similarity measure stands for a high degree 
of similarity and vice versa. 

15 A further embodiment of the image processing unit is arranged to carry out the 

following steps of the method: 

- calculation of the similarity measures between the current image and the previous 
images or a sub-quantity thereof; 

- selection of those previous images whose similarity to the current image exceeds a 
20 predetermined threshold value. 

Thus, via a direct individual comparison it is determined which of the previous 
images are particularly similar to the current image. This approach is advantageously 
combined with the previously described determination of a sub-quantity (by similarity 
comparison with a representative image) in that the individual comparison is carried out only 

25 with the elements of the sub-quantity. The calculation-intensive determination of similarity 
measuires can thus be minimized. 

In the context of a preferred embodiment of the invention, the complete 
electrocardiogram over the duration of at least one heartbeat as well as the instant of 
acquisition of the previous image in relation to this ECG are stored together with each 

30 previous image. Furthermore, the image processing unit is arranged to determine a 

transformation which maps (transforms) the electrocardiogram recorded parallel to the 
current image on the electrocardiogram of a previous image. Using this transformation, the 
relative position of the phase of motion of the current image is determined in relation to the 
phase of motion of the previous image. The transformation used should point-wise associate 
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two different electrocardiograms with one another in an optimum fashion while equalizing 
insignificant variations in time of tiiie physiological potentials. After determination of such an 
optimum transfonnation» a given point in an electrocardiogram can be associated as well as 
possible with the corresponding point the other electrocardiogram. For this association it is 
S important that the transformation takes into account the entire electrocardiogram over a 
period (heartbeat). 

A further embodimmt of the image processing unit is arranged to carry out a 
motion correction between the associated images (current image and a previous image), 
which correction takes into account a motion of the entire body volume. In the context of 
10 medical applications, such overall motions may arise notably due to a change of position of 
the patient. Such overall motions are generally incidental and, therefore, do not contain 
regularities that can be detected during the correction of (natural) motions of the body 
volume. 

The image processing unit may be coupled in particular to a reproduction unit 

15 such as, for example, a monitor or a printer, and be arranged to reproduce the current image 
of the body volume and the associated previous image in superposed form on the 
reproduction unit. Because of the described construction of the image processing unit, a high 
degree of precision is achieved for the correspondence between the current image and the 
associated previous image, so that they can be very accurately superposed. In the context of a 

20 medical application, for example, a physician can then derive fix>m a previous angiographic 
image the vascular tree which is then added to the current image of the body volume which 
shows the instrument (catheter etc.). Because of the hi^ degree of accuracy of the 
association, very exact positioning of the instrument can thus be performed. The previous 
image may especially also be an image firom a preceding step of the same medical 

25 intervention, for example, the image of a stenosis before or during dilatation by means of a 
bulb catheter, or of the position of an already introduced first stent. The superposition of such 
a previous image enables the physician to find the previous position exactly, for example, so 
as to place a stent in the dilatated stenosis or in the vicinity of a first, akeady inserted stent. 

The invention also relates to a device for generating and processing images of 

30 a body volume, which device includes a device for image acquisition, such as notably an X- 
ray apparatus as well as an image processing unit of the kind set fortti. Furthermore, fliis 
device may also include sensors for the acquisition of a signal representing the phase of 
motion, notably an electrocardiograph and/or a respiration sensor (abdominal belt etc.). 
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The invention also relates to a metiiod of associating a current image of a body 
volume, which is subject to motion comprising different phases of motion with one of several 
previous images of the body volume, in which method 

- the phase of motion is detemiined together with the respective (previous and current) 
S images, 

- the current image is associated with that one of the previous images whose phase of 
motion is closest to tibie phase of motion of the current image. 

As has already been described in conjunction with the image processing imit, 
the method offers the advantage that for the association of the current image with a previous 
10 image it takes into account the phases of motion of the body volume at which the images 
have been acquired. Changes of position as caused by the motion of the body are thus 
minimized. 

The method can be extended by determining the distance between the phases 
of motion of the associated images and/or the time elapsed since the last update of the 
15 association between current image and a previous image, so as to reproduce this distance or 
time for the user. Reproduction is preferably performed by means of graphical visualization 
means such as a time bar, the fading of the previous image, or color encoding. 

The invention can also be extended by way of the steps described in 
conjunction with the construction of the image processing unit. 
20 The invention will be described in detail hereinafter with reference to the 

Figures. Therein: 

Fig. 1 shows an X-ray apparatus provided with an image processing unit in 
accordance with the invention, and 

Fig. 2 illustrates the principle of operation of the device in accordance with the 
25 invention on the basis of exenq)lary images. 

The invention will be described in detail hereinafter on the basis of a medical 
application, however, without the invention being restricted in any way to this field. Fig. 1 is 
a diagrammatic representation of an X-ray apparatus with an X-ray source 3 and an X-ray 
detector 1 which are moxmted at the ends of a C-arm (not shown) and form an X-ray image of 
30 the body volume of a patient 2 arranged therebetween. This image is applied as the current 
fluoroscopic image 8 to an image processing unit 5 (in real time). 

At the same time the ECG of the patient 2 as well as a variable representing 
the respiratory cycle is acquired and presented to the image processing unit 5 in the form of 
signals 9. 
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The image processing unit 5 comprises a memory 4 in which previous images 
of the body volume of the patient 2 are stored. Such images may notably be angiogr£4)hic 
images which have bem acquired by means of the X-ray apparatus 1, 3 while utilizing a 
contrast medium and which represent the vascular tree in the body volume in higjilighted 
form. The previous images may also have been acquired by means of other equipment (MR, 
ultrasound, scintigraphy or the Uke). Furthermore, the previous images may also be buffered 
images or image sequences concerning the current intervention which have been acquired by 
means of the X-ray apparatus 1, 3. Images of this Idnd can reproduce in particular the 
position of an instrument, such as that of a catheter which has been introduced into the 
vascular system of the patient and has a catheter tip, or of a guide wire. 

The image processing unit 5 is also coiq)led to (at least) two monitors 6, 7 and 
is arranged to display the current image 8 "live" on both monitors 6, 7 and to display on the 
monitor, sup^posed thereon, one of the previous images derived from flie memory 4. The 
parallel (superposed or separate) display of a previous image serves to facilitate the 
navigation of the instrument in the vascular tree of the patient 2 by the physician. For 
example, a previous angiographic image offers a sort of vascular map (road map), or a 
previous image of the same medical intervention shows, for example, the position of a 
stenosis dilatated by a bulb catheter or the position of a previously placed first stent. In the 
latter cases the previous image assists the physician in repositioning the instrument to a 
previously adopted location. 

For the usabiUty of the previous image it is important that the position of the 
organs and vessels represented therein corresponds as accurately as possible to the situation 
in the current image. For given applications, and in the case of superposed reproduction of 
current and previous image, a precision in the range of a millimeter or even in the sub- 
tnilUmeter range should be pursued. 

The above priecision is acHeved in acc^ 
natural motion of the body volume itself, caused by heartbeat and/or respiration, is taken into 
account for associating a previous image from the memory 4 witii the current image 8. 

In order to enable the heartbeat to be taken into account, the ECG over the 
duration of the at least one heartbeat, during which the previous image was generated, as well 
as the instant of acquisition relative to tiie ECG are also stored in the memory 4 together with 
the previous images. Mapping the entire ECG cycle on the interval [0, 2n\ enables the 
acquisition instant to be expressed as a value from this interval which reflects the relative 
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heartbeat position of a previous image and subsequCTtly serves as a first index of the 
previous image. 

In order to enable the respiration to be taken into account, moreover, a second 
index is provided for the previous images; this second index reflects their relative position in 
the respiratory cycle. The second index is also typically normalized to flie interval [0, 2u]. 
The second index can be derived, for example, from tihie measuring signals of a respiration 
sensor. Altematively, the second index may also be acquired by way of a similarity 
comparison of the previous images with a reference image R which belongs to an extreme 
instant of the respiratory cycle (for example, "deep inhalation"). The second index of a given 
previous image then indicates its similarity distance relative to the reference image R and 
thus reflects the relative position in the respiratory cycle. 

Said reference image R itself may have been selected from the previous 
images. In order to find an image from an extreme phase of respiration from among such 
previous images, for each previous image its similarity measures relative to a series of 
sequential images (over at least two respiratory cycles) can be calculated experimentally. If 
these similarity measures change, for example, periodically with approximately double the 
respiratory frequency, the experimentally considered previous image will be an image from a 
central phase of the respiratory cycle; however, if the similarity measures change periodically 
with approximately the single respiratory fi:equency, the previous image considered belongs 
to an extreme phase of the respiration so that it is suitable for use as a reference image R, 

Various methods for associating a previous image with a current image will be 
described in detail hereinafter. 

Fig. 2 shows a first method of association in accordance with the invention on 
the basis of a diagrammatic representation. The upper row shows the sequence of 
fluoroscopic live images of the body volume, in which a catheter 12 is propagated, on the 
monitor 6 of Fig. 1 . One of these live images constitutes the "current image 8" on which the 
following explanation is based. 

A respective previous image 10a, 10b, ... is superposed on the live images on 
the monitor 7 of Fig. 1 ; these previous images are fetched &om the memory 4 and updated at 
intervals. The previous images may be, for example, angiographic images showing the 
vascular tree in the body volimie. The selection and association of a previous image 10a with 
the current image 8 takes place in three steps according to the first method of association: 

First those images which have approximately the same similarity gap in 
respect of the respiratory cycle as the current image 8, relative to a predetermined reference 
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image R, axe selected £rom Ibe memory 4. To this end, tbe ciot^^ is compared with 

the reference image that is, a similarity measure r between the two images is calculated. 
Analogously, the similarity measures can be calculated between the reference image R and 
all previous images present in the memory 4. The latter has to be performed only once for a 
5 given quantity of previous images, because the measures do not change. As has already been 
explained, said measures may be normalized and be added as a (respiration) index to the 
previous images. Therefore, the amount of calculation work required dxtring operation is 
comparatively small. 

Using the sinoilarity measures, or the indices, a sub-quantity U of tiie previous 
1 0 images can be determined whose memb^ have approximately the same degree of similarity 
with respect to the reference image R as the current image 8. This means that the similarity 
measures of these images lie, for example, withia a window r ± A. If the memory 4 does not 
contain any image that satisfies this condition, the method of association is raterrupted at this 
point. 

15 However, if the sub-quantity U contains at least one element, a second 

selection in respect of the respiratory cycle is carried out in a second step. The previous 
images in the sub-quantity U are then individually compared with the current image 8, that is, 
the associated similarity measures r' are calculated and a sub-quantity V ^ U is determined 
whose images exceed a limit value in respect of the similarity to the current image 8. The 

20 calculation-intensive individual comparison with the image 8 is minimized by the 
preselection of the quantity U. 

Finally, in order to take into account the heart beat, in a third step that previous 
image 10a whose relative instant in the ECG is closest to the relative ECG instant of the 
current image 8 is selected firom the sub-quantity V. For comparison between the 

25 electrocardiograms of the current image 8 and a previous image, a transformation is 

determined, for ^anoiple, by means of a dyaanaic programming algorithm; this transformation 
maps the electrocardiograms on one another in an optimum fashion, thus enabling an exact 
prediction of the phase differences between the striking features (R, S, T lobes) of the 
electrocardiograms. 

30 The described method of association can be modified in conformity with a 

first version in such a mann^ that first a sub-quantity of previous images is detemiined by 
ECG comparison. That one of the elements thereof which best matches the current image can 
then be determined by means of a direct comparison of the similarity to the current image. 
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A second version of the method of association utilizes the described indices of 
the previous images for the cardiac cycle and the respiratory cycle. Via these indices a point 
in a two-dimensional parameter plane can be associated with each previous image. A (usually 
elliptical) proximity can then be defined around the current image which is also represented 
S in the parameter plane; previous images situated in this proximity (if any) then are potential 
candidates for the association. Among these candidates the best association can be found by 
direct comparison of similarity to the current image. A method of this kind offers the 
advantage that it enables the cardiac cycle and the respiratory cycle to be taken into account 
in combined form in the two-dimensional parameter plane. 
10 In a post-processing step (not shown) subsequent to the association, an 

estimate of motion and a correction between ttie selected previous image 10a and the current 
image 8 can be carried out so as to compensate for changes due to a (whole body) motion of 
the patient. 

For the above methods, for example, the histogram energy of the image 
15 differences is a suitable measure of similarity between two images. The images to be 

compared are then subtracted from one anothCT one pixel after the other and the histogram of 
this difference image is calculated. The histogram indicates how many pixels n(G) of the 
difference image have each time a given grey scale value G. The similarity measure r is then 
calculated as the histogram energy which is by definition the square sum off all number of 
20 pixels: 

G 

This definition means that histograms with a concentration of the grey scale 
25 values have a high histogram energy, whereas histograms.with an as uniform as possible 
distribution of the grey scale values over all pixels have a small histogram energy. The 
similarity measure r according to this definition, therefore, has a small numerical value when 
the compared images have a high degree of similarity, and vice versa. A person skilled in the 
art can define altemative similarity measures on the basis of a local correlation or "mutual 
30 information". 

The previous image 10a detemiined can be displayed on a monitor either 
sqparately or superposed on the current image 8. Because the association between the 
respective current image with a stored previous image usually cannot be continuously 
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updated in real time but only (at least) once in every respiratory cycle, the user is additionally 
offered an indicator 1 1 which symbolizes the iqj-to-dateness of the instantaneously associated 
image. The indicator, for example, may change its color or, as is shown in Fig. 2, it may be a 
time bar 1 1 which becomes shorter as the association lasts longer. Fading of the superposed 
image 10a as a function of age of the association can also be perceived very well by intuition. 
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CLAIMS: 



1 . An image processing unit (5) which comprises 

- an input for the signal of a current image (8) of a body volume, the body volume 
being subject to a motion comprising several phases of motion; 

- at least one input for a signal (9) which represents the phase of motion of the body 
5 volume which belongs to the current image (8); 

- a memory (4) in which previous images of the body volume are stored together with 
the associated phases of motion, 

the image processing unit (5) being arranged to associate with the current image (8) that 
previous image (10a) fix)m among the previous images whose phase of motion is closest to 
10 the phase of motion of the current image. 

2. An image processing unit as claimed in claim 1 , which is arranged to 
determine the distance between the phases of motion of the current image (8) and an 
associated previous image (10a) and/or the time elapsed since the last association and to 

1 5 reproduce it for a user. 

3 . An image processing unit as claimed in claim 1 , characterized in that the body 
volume is a biological body volume and the motion of the body volume is caused by 
heartbeat and/or respiration, and that the phase of motion is detected by means of an 

20 electrocardiogram and/or by the signal from a respiration sensor. 

4. An image processing unit as claimed in claim 1, which is arranged to carry out 
the following steps of the method: 

- calculation of a similarity measure (r) between the current image (8) and a 
25 representative image (R); 

calculation of the similarity measures between the representative image (R) and the 
previous images or a sub-quantity thereof; 
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- selection of those previous images (U) whose similarity measm'e relative to the 
representative image (R) lies in a predetermined range (r ± A) around the similarity 
measure (r) of the current image (8) relative to the representative image (R). 

S 5. An image processing unit as claimed in claim 1, which is arranged to carry out 

the following steps of the method: 

- calculation of the similarity measures (r') between the current image (8) and the 
previous images or a sub-quantity (U) thereof; 

- selection of those previous images (V) whose sinailarity to the current image exceeds 
10 a predet^mined threshold value. 

6. An image processing unit as claimed in claim 1, characterized in that, together 
with each previous image there are stored the associated electrocardiogram and the relative 
instant of the image acquisition, and that the image processing unit is arranged to carry out 

15 the following steps of the method: 

- determination of a transformation which maps the electrocardiograms of the current 
image and a previous image one onto the other; 

- detemiination of the relative position of the phase of motion of the current image and 
the previous image, as expressed in the electrocardiogram, by means of the 

20 transformation. 

7. An image processing unit as claimed in claim 1 , which is arranged to carry out 
a motion correction for a motion of the entire body volume between the current image (8) and 
the associated previous image (10a). 

25 

8. An image processing umt as claimed in claim 1, characterized in that it is 
coupled to a reproduction unit (7) and is arranged to reproduce the current image (8) and the 
associated previous image (10a) in superposed form on the reproduction unit. 

30 9. A method of associating a current image (8) of a body volume which is subj ect 

to a motion comprising different phases of motion with one (10a) of several previous images 
of the body volume, in which method 

- the phase of motion is determined together with the relevant images; 
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- the current image (8) is associated with that one (10a) of the previous images whose 
phase of motion is closest to the phase of motion of the current image. 

10. A method as claimed in claim 9, characterized in that the distance between the 

phases of motion of the current image and the associated previous image and/or the time 
elapsed since the last association are determined and reproduced for the user. 
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